Shaping as a Method for Accelerating Reinforcement Learning

be facilitated by rst learning to solve related simpler problems. The term \shaping" itself has been attributed to the psychologist Skinner 7], who used the technique to train animals such as rats and pigeons to perform complicated sequences of actions for rewards. Skinner describes how the technique is used to train pigeons to peck at a speciic spot: We rst give the bird food when it turns slightly in the direction of the spot from any part of the cage. This increases the frequency of such behavior. We then withhold reinforcement until a slight movement is made toward the spot. ...We continue by reinforcing positions successively closer to the spot, then by reinforcing only when the head is moved slightly forward, and nally only when the beak actually makes contact with the spot... The original probability of the response in its nal form is very low; in some cases it may even be zero. ...By reinforcing a series of successive approximations, we bring a rare response to a very high probability in a short time. ...The total act of turning toward the spot from any point in the box, walking toward it, raising the head, and striking the spot may seem to be a functionally coherent unit of behavior; but it is constructed by a continual process of diierential reinforcement from undif-ferentiated behavior, just as the sculptor shapes his gure from a lump of clay. (Skinner 7] pp. 92-93) The phrase \...reinforcing a series of successive approximations..." expresses the essence of shaping. Given the task of training an animal to produce complex behavior, the trainer has to be able to (1) judge what constitutes an approximation to, or a component of, the target behavior, and (2) determine how to diierentially reinforce successive approximations so that the animal easily learns the target behavior. Unfortunately, neither of these two components of shaping have been formalized rigorously in the psychology literature, even though shaping is widely used both in psychological studies and to train pets and circus animals. Staddon 8], for example, observes that the trainer often has to rely on an intuitive understanding of the way the animal's behavior is generated when determining which behavioral variations are precursors to the target behavior and how to reinforce these precursors. Variations in the behavior of individual animals also must be accounted for when making these judgements. mass of the pole, shortening the pole, and …